Overview
Neural Network

Custom Control 1.0

Overview

This custom control implements a feed forward multilayer neural network device with training by error backpropagation.

It allows you to define,train and use neural network devices in your programs easily. It also supports save and load networks to disk for later training or use.

Properties        Methods        Events       Error Codes

Comments:

Input and output values are float numbers ranging from 0.0 to 1.0 for inputs and 0.1 to 0.9 for outputs.

The control uses a 'standard' Action property to control the net.

Net Use:

The usual process in implementing neural nets has 3 phases, Design, Train & Use.

Design phase.

The Design phase is simple, set the topology of net (properties I, O, and optionally H0 and H1) and set Action property to 1 (Create). This creates and initializes the network in a random state in memory.

Train phase.

In the Train phase, you must define a trainig set of examples , set the Rate, Momentum and MaxError properties, and repeteatedly set the Action property to 6 (Train), till the Error property is small enough for your needs.

To support the trainig set, the control uses a ListBox like mechanic. You have a List(), ListCount, and ListIndex properties, and the AddItem, RemoveItem and Clear methods, like a normal listbox. (Clear only allowed in VB 3.0 compliance). List values are string values especifiying the inputs plus the outputs values, separated by a espace, tab or linefeed. The decimal point, must be expressed with the '.' (point) character. For instance, in a network with 3 inputs and 2 outputs, a valid string could be '.125 .9 .1 .55 .9'. Alternatively, you can set the Inputs() and Outputs() properties and set the action to 2 (AddItem), this adds the current inputs and outputs values at the bottom of the List() property.

Once created or trained, the network configuration can be saved on disk for later recall. The file name is especified in the Filename property, and setting action property to 4 (Save) or 5 (SaveTrain) causes the memory image to be written on disk. Note that a network saved with the Save action, cannot learn further, but consumes much less memory than a network saved with SaveTrain action. Remenber that no save or load of List() values is done, the process to store the trainig set is completely left to the user.

Use phase.

The Use phase implies to get in memory a trained network. This can be done in two ways: a) you already have a network created and trained (via action Create and Train); b) You set the Filename property and use the Load action =3.

Once in memory, all you have to do is set input values with the Inputs() table property, and set the action to 7 (Work), or alternatively use the Refresh method. The result is in the Outputs() table property.

Training considerations:

Training a neural network needs a massive floating point use. The objective is to minimize the error of an unknown function defined by input-output couples in the training set, starting at a random network state.

Several cases can occur depending on training set values, initial random state, network topology, and rate and momentum values. They can be grouped in two major cases.

1 - The net can reach a global minimun error.

In this case, network topology is good enough to satisfy the trainig needs, although the training phase can be slow, and the network can need a very large number of trainig cicles.

2 - The net cannot reach the global minimun error.

In this case, network topology can be good enough, but initial starting point and rate and momentum values leads to a local minimun, and the network gets trapped in it. Or worse, the net topology is too simple and cannot learn the training set, no matter the number of trainig cicles used.

As a general rule of thumb, two behaviors can be detected. The net error does not change, no matter the training cicles we use. Or the net error oscillates 'ad aeternum' between two or more local minimuns.

Avoiding local minima.

To try to elude the local minima problem, remember that if a deterministic function is defined by the training set, the net can learn it, but if we are trying to learn a non deterministic function, the error precision becomes decisive. Ex. if the trainig set consist in two examples, one, where input vector A leads to output vector B, AND two, where the same input vector A leads to output vector C, the error cannot be smaller than a difference function between vectors B and C.

Use small rate values, this usually leads to a large number of training cicles, but alleviates the probability to get stuck in an oscilative behaviour.

Choosing a network topology.

Start trying a three layer net with few (ex. 2) hidden neurons, if the net cannot get the global minimun, increase the number of hidden neurons.

If global minmun cannot be reach, maybe we are trying to optimize a function with discontinuities, try with a four layer network, starting with a few hidden neurons in both layers, and increase the number of hidden neurons as needed. Usually the hidden layer 0 (H0 property), should be bigger than hidden layer 1(H1 property).


Propertys
Properties

I

H0

H1

O

Inputs

Outputs

Action

Filename

List

ListCount

Listindex

Error

Rate

Momentum

MaxError


I Property

Type int

This property indicates the number of input neurons of the net.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


H0 Property

Type int

This property indicates the number of neurons in the first hidden layer.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


H1 Property

Type int

This property indicates the number of neurons in the second hidden layer.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


O Property

Type int

This property indicates the number of output neurons of the net.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


Inputs Property
Inputs() Property

Type float

This is an array property to access to the input neurons of the net. Legal values are form 0. to 1. . Legal index values are from 0 to I property value -1.

Properties


Outputs Property
Outputs() Property

Type float

This is an array property to access to the output neurons of the net. Legal values are from 0.1 to 0.9 . Legal index values are from 0 to O property value -1.

Properties


Action Property

Type int   (write only)

By setting this property, the user controls the net behaviour.

Available actions:

0 - None

1 - Create

2 - AddItem

3 - Load

4 - Save

5 - SaveTrain

6 - Train

7 - Work

8 - Destroy

Properties


Filename Property

Type stringz

This property holds the filename to be used in the Load , Save, and SaveTrain actions. No checking of valid filename is done, if the user sets an invalid filename, an error will be trigged in the action process.

Properties


List Property
List() Property

Tipe string

This is a property array to hold the trainig sets. Although it is an string property, at runtime, strings are parsed and float values are stored, or alternatively, stored float values are writen to return a string value. Due to the conversion process, it could be  possible to read a string value different than the written one.

Legal values are a string with input float values plus output float values. Ex. in a net with 3 inputs and 3 output, a valid string could be "0. 1. 0. .1 .5 .9".

Note that values must conform with the legal ranges for inputs() and outputs(), otherwise, an error is trigged.

Note also that the '.' (point) character is used to indicate the decimal point, and space, tab or linefeed characters are used as token separators.

Properties


ListCount Property

Type int

This property holds the count of trainig sets presents in the net. Read only.

Properties


ListIndex Property

Type int

This property is similar to the standard ListIndex property of ListBoxes, but in this control, setting this property to a legal value (below ListCount property value), causes the net to present the List() property value referenced by ListIndex, in the Inputs() and Outputs() properties.

Properties


Error Property

Type float

This property holds the error value found in a Train action. This value is computed as a standard mean square error between output vector values and target vector values.

Properties


Rate Property

Type float

This property holds the rate of gradient correction used to modify conexion values in the trainig cycle. It is usually set between 0. and 1.

Even if this isn't the standard method, you can change its value between trainig cycles.

Properties


Momentum Property

Type float

This property holds the momentum (inertia) constant used to correct the network connexions. It is usually set between 0. and 1.

Even if it is not the standard method, you can change its value between trainig cycles.

Properties


MaxError Property

Type float

This property holds the max error allowed in a train action. If after train is done the Error property is bigger than this value, the network corrects its weights, if not, training is ignored, and a Trained event is fired, so its a waste of time to keep using the Train action if Error is lowest than this value.

Properties


Network Topology

The neural network custom control implements a multilayer design. Each layer is fully connected to the next, and an (invisible) bias neuron is added at each layer.

To keep the complexity at the lowest level possible, this version does not support other interconnecting posibilities.

You can select between a 2, 3 or 4 layer design, setting the  I , O , H0 , and H1  properties.

To select a 2 layer design, leave the H0 and H1 properties to zero.

To select a 3 layer design, leave only the H0 or H1 properties to zero.

To select a 4 layer design, set the I, O, H0, and H1 properties to a nonzero value.

Memory considerations & Limitations.

This simple scheme, does not imply a limitation of network capabilities, it is assumed on papers that any degree of complexity can be achieved with a 4 layer net.

You can set any value in the I, O, H0 and H1 properties, but  the number of conexions between two adjacent layers must be less than 8000. According to the conexion scheme explained before, this number is computed as:

   (neurons_in_previous_layer + 1) * neurons_in_next_layer

This limitation arises from the fact that any connexion is a 8 byte value in work mode, and three 8 byte value in train mode. To elude the use of huge pointers, and speed operation in this version, any layer vector must fit in a segment area, so this leads to the 8000 limit.


Trainnig
Trainig

The net uses a 'standard' gradient/momentum method to implement training.

It consist in two phases:

Phase One.

Traing set (List() property values) is presented to the net and the output values are compared with expected target values. With this info, gradient vectors and Error property is computed, and if Error is bigger than MaxError property value, the net activates the phase two, otherwise, the control fires a Trained Event to indicate that no correction will be applied to connexions.

Phase Two.

In this phase, the net uses the gradient vectors computed in phase one, the previous momentum vectors, and the Rate and Momentum property values, to correct the connexions to approximate to solution and save the actual momentum vector for the next trainig cycle.


Trainnig Sets
Trainig Sets

In work time, a neural network device, is capable of reproducing in response to its actual inputs values, the output values 'learned' in the train phase . To support the trainig set, or collection of input-output couples, this control implements a ListBox like mechanich.

Any couple of inputs-outputs values is stored as an entry in the List() property. To support the insert , remove or access of items (examples), three properties ( List(), ListCount, ListIndex),  three methods ( AddItem , RemoveItem, Clear ), and an Action value (AddItem (2)) are implemented.

In any Train action, the entire set of List values are considered, but you can change the List contents between Train actions.

No load or save of List() values is supported, the user must implement his own system to store the trainig set.


Methods

AddItem

RemoveItem

Clear

Refresh


AddItem method

This method is similar to the standard AddItem method of ListBoxes, except for the format of the string value, see List() property for details.

Methods


RemoveItem method

This method is similar to the RemoveItem method of ListBoxes.

Methods


Clear method

This method is similar to the standard Clear method of ListBoxes. Deallocates the actual trainig set. and set the Listcount property to 0, and ListIndex property to -1.

Avalilable only in VB 3.0 compliance systems. VC++ supports only VB 1.0 compliance !.

Methods


Refresh method

This method is used to activate the network. It is equivalent to set the Action property to 7 (Work action).

Methods


Create action

Value 1

This action checks the I , O, H0 and H1 propeties, and if legal, allocates the net, inites the connexions at a random state, and zeroes the momentum vectors. Once allocated, the net topology cannot be changed, you must use the Destroy action to free resources and return the net to a not created state.

Actions


AddItem action

Value 2

This action adds the current input and output values ( Inputs() and Outputs() properties) to the bottom of List() values. It is implemented to facilitate the trainig set definition.

Actions


Load action

Value 3

This action uses the Filename property to load a network previously saved with the Save or SaveTrain action.

Actions


Save action

Value 4

This action uses the Filename property to store in disk the current net configuration, excluding training vectors. This results in a small image, but  note that networks saved with this options cannot learn further.

This action is recommended when the net has been completely trained and no further learning is required.

Actions


SaveTrain action

Value 5

This action uses the Filename property to store actual network configuration in disk, including trainig vectors. Nets saved with this action, can be loaded later, and continue the learning process.

Note that although the trainig vectors are saved, the trainig sets (List() property values) aren't saved. The user must implement his own processing to save and load the trainig sets. This is done to flexibilize the control usage, allowing the user to integrate the control with external data.

Actions


Train action

Value 6

This action starts a trainig cycle. Once finished the Error property indicates the Error amount found, and if this Error value is less than the MaxError property, a Trained Event will be fired.

Actions


Work action

Value 7

This action activates the network. Actual Inputs() values are presented, and  Output() values are computed.

Actions


Destroy action

Value 8

This action deallocates the net resources, free List() values, and returns the net to an initial not created state. Only in this state can be changed the network topology.

Actions


Events

The control implements a unique event in response to the Train action.

Trained Event

If the Error property falls below the MaxError property in response to the Train action, the control fires the Trained Event to indicate that no further correction will be applied to connexions. This event has no params.


Error Codes

The following set of error codes are supported.

20000 Network allready created, Use Destroy Action.

20001 Invalid Value.

20002 No Available Memory.

20003 Network not created, Use Create Action.

20004 Reserved.

20005 Too much connexions, value not allowed.

20006 Cannot open file.

20007 Cannot write file.

20008 Cannot read file.

20009 Incompatible Neural Network version.

20010 Input or Output len < 1.

20011 Listcount<1, nothing to learn.

20012 Network cannot learn.